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This group of reports constitutes the first series of 
observations designed to document the experiences of six state 
agencies (Delaware, Oregon, Michigan, Virginia, Pennsylvania, and 
Mississippi) which are developing model program evaluation/management 
support units that will affect the development and implementation of 
policy in federal/state programs for vocational rehabilitation (VB) . 
In the first report the failure of program evaluation in 
rehabilitation is traced to the fact tfcat counselors are not 
neccesarily evaluators, to the existence of vague and ambiguous VB 
goals, and to faulty evaluation methodology. Efforts in the 
development of VB program evaluation criteria are discussed in the 
second report on the relationship of program evaluation and VB 
agencies. In the third and fourth reports the Pennsylvania and 
Mississippi model evaluation units document their experiences in the 
formation of their model units. Suggestions and forms for managing 
information requests dealing with VB evaluation are provided in the 
fifth report. Two book reviews (the first on Joseph S. Wholey's 
Evaluation: Promise and Performance and the second on Scarvia 
Anderson and Samuel Ball's The Profession and Practice c£ Program 
Evaluation) constitute the final two reports. Appendixes include a 
bibliography of program evaluation literature and a list of 
periodicals and journals relating to rehabilitation and evaluation. 
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FOREWORD 



.J^ Rehabilitation Services Administration (RSA), initiated 
DevelSn!Sn^ Program and Policy Systems Through 

Development of Model Evaluation/Management Information Support 
l^il Zt ^^f^^ Departments of Vocational Rehabilitation in 

1978. These evaluation demonstrations have produced some reward- 
nvPr^Jhr^^"?^!: ""^^^ publication is the first of twelve scheduled 
over the next three years. Each edition will feature articles 
written by the principal investigators of the model units, federal 
personnel, staff of firms and agencies under contract to RSA, and 
evaluation experts in the rehabilitation field. We will devote 
particular attention to providing information on achievements and 
barriers experienced by the model units over their three year 
existence. We will also keep readers posted on trends in program 
eJaln^^ton rehabilitation, initiatives underway to build state 

i= c^P^city, and other general and technical evaluation sub- 
jects as we progress. In light of the demonstrated efforts of the 
fo^^i.r "T^"^ °f ^^^^"^ accomplishments will have implications 

2aJ program evaluation at the national and 

state levels of the Vocational Rehabilitation Program. 



James E. Taylor, Ph.D. 
Project Officer, RSA 
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PREFACE 



The priinary purpose of this series of reports . ) cbcument the experiences 
of six state agencies that are developing systems .jact upon the develop- 
ment and inplannentation of policy in the Federal/state program for vocational 
. rehabilitation. 



On October 1, 1978, the Rehabilitation Service.^ Administration (RSA) 
entered into a oontractual agreanent (one year dur-^ )n with the option for tvgo 
additional years) with six state Vocational Rehabii ^.tion (VR) Agencies to 
develop model program evaluation/management information support units. The 
contracts were awarded on the. basis of the technical merit of proposals sub- 
mitted in response to a Request For Proposals (RFP) . The states receiving 
the contracts were: 



Delaware (a sroall general agency) 
Oregon (a small general agency) 
Michigan (a medium sized general agency) 
Virginia (a medium sized gcaieral agency) 
Pennsylvania (a large general agency) 
Mississippi (an agency for the blind) 

The objectives for the MDdel Evaluation Units (MEUs) were to: 1) develop a 
model in vAiich conprehensive program and policy systems were linked by appro- 
priate evaluation data; 2) field test and evaluate the effectiveness of the 
(revised) Federal Program Evaluation Standards and the Facilities Information 
Systan for VR agency management; 3) build new evaluation capacity viiich can 
be generalized to other states; and 4) develop linkages for a within-state 
agency and between-state agency network for oomnnunication, dissemination and 
utilization of evaluation topics, with special enphasis on developing and 
testing the Model Evaluation Units. 

In order to meet these objectives, the MDdel Evaluation Units were required 
by their first year's contract to perform the following tasks; 

1. Plan and organize the MDdel Evaluation Units; 

2. Initiate and establish continuing working relationships with 

associated organizations, contractors and university re- 
sources; 

3. Administratively rnonitor the organization of the Evaluation 

Unit; 

4. Field Test the new (revised) Federal Program Evaluation 

Standards; 

5. Field Test the Vocational/Medical Pacilities Information 

System; 

6. Develop New Evaluation Capacity; 

7. Assure dissemination and utilxzation of products; 

8. Evaluate the project, and 

9. Suhmit yearly and final reports. 



V 



On October 1, 1971, RSA, after cxmpetitive bidding, awarded a c3ontract 
to the West Virginia Research and Training Center (WVRTC) to coordinate the 
activities, of MEU develoExnent. Ihe primary functional responsibilities of 
the WyPTC are to provide ooordination, promote technical assistance, nonitor 
activities, develop models, conduct evaluations, ard prepare articles and 
other materials for dissemination. The specific tasks of the WVRTC for the 
most part coincide with those of the MEUs; however; additional tasks include 
the develoEinent of a regional office (RSA) model for the lose of evaluation 
data generated by state VR agencies and the developiTBnt of a set of manu- 
scripts that will be instructive to other agencies who want to incorporate 
MEU concepts/products into their program evaluation units. 

IWo other contracts have been awarded by RSA to provide specialized 
assistance for Tasks 4 and 5. Task 4, to pretest the New (Revised) Federal 
Program Evaluation Standards, will be facilitated by Berkeley Planning Asso- 
ciates (BPA) EPA developed the new standards urder a previous contract with 
RSA. Iheir involvement with the MEUs includes the pretesting and ref inenent 
of the proposed performance, procedural, and project standards. Activities 
will include designing instrurents for data gathering, training the states 
in instrument use, coordinating the pretest, analyzing the data, revising 
standards as necessary, and preparing an irrplementation plan. 

Walker Associates, (WA) , under an agreenent with the National Associa- 
tion of Rehabilitation Facilities, will be providing specialized assistance 
to the MDdel Evaluation Units in the conduct of Task 5 - Pretest the Facilities 
Information System (FIS) . Ihe FIS was developed by WA under a previous con- 
tract with RSA. Ihe major tasks that WA will be conducting include training 
for MEUs in the use of the FIS, pretest iirplementation assistance, monitoring 
and evaluation of the pretest experience, revision of the system where neces- 
sary, and the developsnent of recommendations for nationwide irrplementation. 

The reports of this project will contain (a) issue papers that raise 
questions and suggest answers in generic evaluation problems; (b) descriptive 
reports of the methods employed and resialts of particialar evaluation studies; 
(c) "how to do it" articles; and (d) reports doejmenting the experience of the 
various staffs. These reports will be published quarterly for a period of 
three (3) years. 

This first series of observations contains an issue paper by the West 
Virginia Research and Training Center that deals with the determination of 
evaluability as the first step in program evaluation. An historical 
perspective of evaluation in VR is the main thrust of an article by the Vir- 
ginia MEU. The Pennsylvania and Mississifpi MEUs document their experiences 
in the formation of their model units. The Oregon MEU provides a "how to do 
it article" on dealing with information requests. IWo book reviews are also 
included, as well as three informational appendices. 



Dr. Richard A. Nida 
Project Officer 
August, 1980 
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TRACING THE FAEIUFES OF PIW3RAM EVALUATICN 



IN VCXKnCNAL FEHABILITAnON 



RICHATO A. NIDA, PH.D. 
WEST VIRGINIA RESEARCH & TRAINING CENTER 



^teny social services prograonns that flourished during the sixties fouiKi 
that the seventies brought a cessation to growth and! in many instances a 
redu c tion in avziilable program funds. In vocational rehabilitation (VR) , 
1972 marked the end of real growth (when the total allocation is corrected 
for inflation) . The subsequent reduction in allocations for progr a m 
parallels the increased OTjiiasis and funding for prog ra m evaluation (PE) 
activities. 

The Rehabilitation Act of 1973 (P.L. 93-112) provided the basis for the 
exte n ded requirements and cc^bilities within state agencies for VR. Funding 
for program evaluation activities at the central office of the tehabilitation 
Servix:«s Administration increased to $1 million in 1974 and to nearly $2.5 
million in 1978. Precise assessments for monies expended at the state level 
are not readily available, but it is safe to say that state expenditures at 
least equal the Federal ones. 

The very real decrease in program funds, coupled with a growir^ 
sk^>ticism about the iitipact of services, in part forced the growth of program 
evaluation.. Presumably, in a time of scarce resources, evaluation wDuld pro- 
vide the necessary infontation about program operation that would allow 
managers and policymakers make decisions to iitprove the quality of ttose 
programs. To date, the pronise of program evaluation is yet to be realized. 

Schniidt, Scanlon, and Bell (1979) noted that program evaluation has not 
led to successful programs and has produced little information of utility or 
even interest to managers aixi policymakers. What, then, has led to the 
failure of program evaluation? 

Stock Answers for VR PE Failures 

The most often listed reasons for the lack of effectiveness of program 
evaluation in state VR agencies have been that PE is "new"; and that the 
evaluators are, in most instances, former counselors vto have not ted formal 
treiining in evaluation techniques. 

How New is "New"? 



Prior to 1960 a reference to program evaluation was difficult to find 
in any social services lit>erature. 



Evaluatim, as v^ell as systenatic planning, in the state/tfederal rehabi- 
litation program was first begun with the statewide planning and the 
continuing statewide studies initiated by Federal legislation in 1965. This 
legislation authorized two-year planning grants for state rehabilitation 
agencies to evaluate rehabilitation programs and to delineate the steps 
necessary for the developnent of adequate vocational rehabiUtation resoiaroes 
and programs within each state. Additionally, states were required to co- 
ordinate rehabilitation planning with other statewide planning activities and 
with citizen involvement. 

•Ihe amendments the 1968 legislation specifically required state 
agencies to evaluate the services they were providing. In addition to the 
requirement. Federal funds for evaluation of VR programs were first made 
available in 1968. The Rehabilitation Act of 1973 (P.L. 93-112) further 
extended the requirements and capabilities for evaluation within Vocational 
Rehabilitation Agencies. This law contained severed provisions that speci- 
fically impacted on evaluation, but Section 401 is primarily of interest. 
Section 401 details the reporting requiranents of the Secretary to Congress 
and gives instructions that "evaluations shall be conducted by persons not 
initiediately involved in the adnunistration of the proqram or proiect 
evaluated." ^ 

Section 401(b) of the Act required the Secretary to "develop and publish 
general standards for evaluation of the program and project effectiveness in 
achieving the objectives of this Act." Further, it specified that the 
Secretary shall consider the esctent to v*iich such standards have been met in 
deciding v*ether to review supplemental financial assistance authorized under 
any section of the Act. Interim evaluation standarxis were published in the 
Federal Register on July 2, 1974, and final regulations were published in the 

^^stBX on December 19, 1975. Presently the evaluation standards 
have been revised and are awaiting field testing as part of the contracts of 
the six itodel units. 

Program evaluation is "new" to VR. It is new, period. IV^enty years is 
not a long time frame for a discipline to develop. The first stock answer 
does appear to explain some of the PE failures to date, but that reason be- 
comes less viable with each passing day. 

Oounselors Aren't Necessarily Evaluators . 

When VR agencies were first faced with the requirement to staff a 
program evaluation unit, many of them placed either former or actively 
practicing oounselors into the pDsition(s) . A survey of evaluators in state 
VR agencies conducted just prior to the enactment of the Rehabilitation Act 
of 1973 (Ridge, 1973) indicated that most evalviation personnel did not have 
formal training in either the theoretical or operational cornponents of 
evaluation. The background of most staff members was grounded in rehabili- 
tation rather than evaluation. Spaniol (1975) also indicated that evalu- 
ation personnel were uncomfortable with present skill levels and needed 
training in evaluation methodologies. 
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The initial skill level of these cx)unselors turned evaluators was irdeed 
inadequate, but advanced training and on-the-job skill acquisitions has 
resvdted in increased evaluation expertise at VR agencies. Additionally, 
many agencies are now recruiting evaluation specialists f riDm outside the 
agency to further build evaluation capacity. Again, the stock answer v*iile 
operative is losing its explanatory power. Other factors must then be the 
source of the continued failure of program evaluation to effectively irrpact 
VR program and policies. 

Potential Sources of Evaluation Problems 

One need not look far to find potential sources of evaluation problems 
in state VR agencies. IWo examples include the VR program itself and the 
evaluation methodology most often in VR settings. 

VR Program 

Van Maanen (1979) identified the basic evaluation question as "Tto what 
extent is the program actually reaching its goals?" (p. 43) . This kind of 
goal attainment model is the most often used in VR agencies (Oook and Cooper, 
1978) . As many writers have noted program goals and objectives are often 
vague, ambiguous, changeable, difficult to specify, and even inoonpatihle 
with each other within a given program. 

Given that the program evaluator rarely sets the goals and objectives 
of the agency, difficulty in evaluating the extent to v*iich the program is 
meeting a set of vague, ambiguous, chiangeable, difficult to specify the 
potentially incarpatible goals is apparent. Is it any wonder then that the 
VR program and evaluation activities have proceeded essentially on 
parallel tracks without impacting each other? 

If a primary purpose of evaluation is to contribute to the decision 
maJdjig process (Weiss, 1972; Spaniol, 1977), then the clear specification of 
the program and its goals and objectives is a necessary precondition to 
conducting a meaningfial evaluation. An additional precondition listed by 
Rutman (1977) is a rationale that links the program to its goals and 
objectives. Too often the administrators of the state VR program and the 
VR evaluator have failed to assure that these necessary preconditions were 
present before proceeding with an effective evaluation of a program. 

The heavy ertttesis placed i;pon goal setting and specification in this 
docunent does not preclude program benefits occuring either as a resialt of 
unintended consequences or unexpressed goals. In fact other evaluation 
approaches such as a systems approach or goal-free evaluation argue against 
the goal attainment approach most frequently used in VR as being inadequate. 
The inplication for VR evaluators is thus to identify unexpressed goals and 
unintended consequences and incorporate them into the goals attainment model. 
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Evalijation MethodologY 



Specific methods of conducting evaluation studies are not discussed here- 
SrS; a nethod for determining if the above itentioned prSnSSSta^T ' 
^^t"^""^ - brief This methodology is kno«S as "evaluabiSS 
tffS^^' evaluation publication of the General Accounting Office 

^ffl^?^^^03r^]s^Ev^^^on: A Q^list Appg^f ig^) 
^g^^^g^^^^ous E^I^FThat a program's evaluabili ^hSlird mac3e 
SiLiJ^ri^ process; however, itost state VR agency 

evaluators bypass what is logically one of the niDst iiiportant firsHSps. 

nf 9iown out of the work of Dr. Jos^ Wholey 

of the Urban Institute who conducted must of the cSevelopnental work for^ 

So^^t'^-'^J^f ^^-^ ^l^y (1977)' Evaluation : PrcSIe' anT^ 
Performance (reviewed later m another section) T'BSt^H^inniJT^— 

."gg^^^Sl^^^g^t explores the objectives, expectations, 
^ -nformarion needs of program managers and policy-makers; explores 
program reality; assesses the likelihood that program activit^ will 
achieve measurable progress toward program objectives; and assesses 
tne extent to vAich evaluation information is likely to be used 
by program management. The products of evaluability assesatent 
are: (i) a set of agreed-on program objectives,! important side 
effects, and performance indicators on which the program can 
realistically be held accountable; and (2) a set of evaluation/ 
itanagement options which represent ways in which managa«?jit can 
Change program activities, objective, or uses of information in 
ways likely to iirprove pgoram performance (p. xiii) . " 

rv^n^rSfff iJ' °? f^orts involving evaluability assessments have 

occurred at the Federal le/el. rtiis is also true in VR. 

Exarrple of an Evaluability Assessment in VR . 

The Oommissioner of the Rehabilitation Services Administration (RSA) 
recently asked the Office of Technical Assistance of the Assistant Secretary 

f Evaluation within the former Departmsnt of Health, Education, 

and Welfare to conduct an evaluability assessment of the RSA program. The 
EA will, through a series of interviews with representatives of public 
interest groups, governmental control agencies, congressional coitmittees, 
policymakers at the department level, RSA central and regional office 
managers, and state and local VR agency personnel explore: 

1. the program objectives, expectations, and assuitptions of 
policy-makers and managers in charge of the program; 

2. the objective of political groups; 

3. the extent to which management's program objectives have been 
defined in measurable terms; tlie program activities actually 
under way; 



4. the likelihood that p ro c)rdn i activities will achieve measiarable 
progress toward program objectives; 

5. the lately xipe of informatixan in p rog id iu perfbrmai^ce; and 

6. options for change in pro gran t activities, objectives, and 
i:ise of informtion that could enhance p rog r am performanoe. 

HhB results of the evaluability assessment will determine vdiether 
program evaltiation can be used to alter the perfonnance of the program. An 
affirmative response would indicate that BSA top management can be held 
responsible for agency performance in a logical and meaningful fashion, 
Ihe program will have established its evaluability by meeting the pre- 
conditions of a clear expression of program, pro gra m goals and objectives, 
and a rationale between p rogran and goals and cd)jectives. A negative 
response to the evalxaability assessment could, of course, require the 
redefinition of the program, its goals, or the rationale that links the 
two. 

State agencies for VR need to follow the lead of RSA in this respect 
and initiate evaluability studies of their own p ro gra n s to insure that 
time is not lost in conducting evaluations that do not have a reasonable 
e3(pectation of infacting \jpon, pr o gidm performnce. Bie future failures or 
sixxsesses of program evaluation may very well hang in the balance. 
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pgo^flM EvaimriCN and vocaticnal rehabilitation agencies 

W.H.BRCWNFIELD & JOHN E. HAYEK 
VIPGINIA DEPARTMENT OF REHABILITATION SERVICES 



Interest in program evaluation is a relatively new develoEitent in the 
social service field. Schulberg and Blaker (1969) ijidentify post Wbrld War 
Ti Q^oA ^ u ° P^^?9^ evaluation in Public Health. Rossi and WUliams 
t±y/^) see the major mpetus for program evaluation stenming fron social 
action programs of the 1960's. Cbncems for program evaluation has arisen 
tor various reasons, of which most oornDnly cited are accountability of hu- 
man service organizations, and a questioning of the value of new programs. 

Ihe purpose of this paper is to look at acre of the literature of proqram 
^^^^J" ^^^f^f^' highlight the major problems facing program evalm3on, 
discuss the applications of program evaluation to vocational rehabilitation 
i^ Trif^f ^ ?^.?°"^i^^ the implications of program evaluation within state 
and Federal rehabilitation progrartming. 

Progiam evaluation has been defined as: 

. . .a systematic continuous process of providing 
information abaut the value or worthv^leness 
of a program for purposes of decision making. 
(Spaniol, 1975, p. 2) 

Ihis definition has several concepts the authar etiphasized to show that 
program evaluation provides deliberate, on-going, planned assistance to deci- 
sion imkers. 

Programs requiring evaluation may be as stall as a unit serving one 
disability group or as large as the agency. The iitportant thing is not the 
size of the program, tut the type of information needed to make informed 
decisions. Thus, at any level program evaluation can be arployed for a 
number of reasons. Fbr us the most uiportant is its use in decision making 
ooncemuig (1) program installation, (2) program oontinuaaon, and (3) pro- 
gram modification. ^ 

Information for program installation is considered to be planning 
information. It includes such things as needs assessnent, cost estimates 
and operational feasibility. Information fron evaluations for program contin- 
uation IS the traditional view of evaluation. It is the evaluation that is 
concerned with outputs and products of a program. Another name for this type 
Of evaluation is a perforrrance audit. Finally, information for program mDdi- 
tication generally emphasizes processes, and managers may consider this type 
of evaluation as a managenent audit. 
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A major difficulty in program evaluation is t±ie lack of utilization of 
results. (Anderson and Ball, 1978. Horst and Others, 1974. RDssie and 
Williams, 1972) Ohere appear to be two sources for this difficiolty; one is 
management based and the other is based in evaluation methodology. In either 
case, if the results are not used the evaluation effort is of little value to 
the agency or manager. 

Management-based lack of utilization of results is seen as a critical 
prctolem cause by a lack of program definition (vague objectives) , a lack of 
the use of clear logic (unjustified assumptions) , or a lack of management 
(management unable or unwilling to lose results) . (Horst and Others, 1974) 
RDssi and Williams (1972) see the conceptual application of evaluation to 
social services as the difficulty. 

The second source of difficulty in utilizing evaluation results begins 
with methodology and includes management problems. Ihus, evaluators must 
face methodological problems (statistics, sairples, control groups, variable 
specifications) ; bureaucratic problems (manager/evaluator conflict) ; politi- 
cal problems (technical ability within the organization to conduct ocxrplex 
studies) . (RDssi and William, 1972) 

The difficulties mentioned here will, at some point, have to be dealt 
with - to ignore the ooitplexities of methodology, the difficulties concerned 
with management and the realities of politics would i^ender only lip service 
to any program evaluation effort. 

As mentioned earlier, program evaluation is a relatively new enphasis 
in the social service field. For vocational rehabilitation it can probably 
be considered still in its infancy. Pressures to f induct program evalx^ation 
in rehabilitation have been both external and inter .lal. Ihe enphasis on 
increased accountability in recent legislation is a major external pressure. 
Idealism beliind programs must new be sipported by more concrete measures to 
show program worth and justification. Programs with social value must be 
quantifiable. If the program depends on emotional appeal for support it must 
also be shown hew its intentions are being acooitplished. Internal pressure 
is viewed as stemndng from the need for more information xjpon v^ch to base 
decisions, increased enphasis on policy analysis, increasing professionalism 
at all levels of the rehabilitation system, and the interest in evaluating the 
outcome (inpact) of rehabilitation services. (Spaniel, 1975) 

In response to the pressures for program evaliaation (PE) , the early 
1970 's was a time of developing units, identifying personnel and beginning 
initial evaluation studies. (Miller, Lee, et al, 1977) Within the VR program 
several difficulties in program evaluation inplementation were identified. 
These include: the lack of appropriate staff training; the lack of conceptual 
framev^ork for program evaluation; the lack of methodologies related to outoane 
evaluation; a tendency to isolate program evaluation from agency program 
development activities; data quality and consistency; inability to specify 
outcomes; and the need for better measures of outconne. (Miller, Lee, et al, 
1977) 



Vocational rehabilitation agencies, in trying to meet the intent of 
legislation as it pertains to evaluation, have themselves identified much 
of v*iat has been discussed. The Rehabilitation Services Administration (RSA) 
initially attenpted to guide PE focus in three major areas: eligible popula- 
tion, process and outcome. (Rubin, 1974) Within states the enphasis has been 
on information for planning, monitoring and program revision. (Crystal, 
Harrison and Miller, 1978) It seems clear that program evaluation in general 
and within the rehabilitation oamiunity have oonnon goals and difficulties. 
The latter part of this paper will review how evaluation has evoloved in VR, 
what is presently happening, and, we hope, provide sore insight on future 
directions of PE within VR. 

Ihe 1965 Amendment to the VR Act provided scxne of the irrpetus for 
evaluation in VR. Further iirpetus came with the Anendments of 1968 vAiich 
outlined requirements for evaluation efforts on an ongoing basis. Ihe RSA 
then asked states to describe in their Program and Financial Plans for 1973 
v*iat evaluation efforts they were pursuing. But probably the nost important 
"motivator" to state VR agencies' evaluation efforts cane with the tehabili- 
tation Act of 1973. 

It was in this Act that the General Evaluation Standards were mandated. 
It was at this point that some rather specific evaluation efforts were re- 
quired by RSA from the state agencies. Ihus, probably for the first tiire in 
the 60 plus year history of VR programs, some specific and discrete evaluation 
data and information were required by the Federal agency. However, no 
additional funding to the agencies was provided. As a result, vMle all state 
VR agencies were involved in program evaluation, the sojiiistication of the 
evaluation effoirts ranged from sinple casework reviews to elaborate evalua- 
tion methodologies. Program evaliaation meant different things to different 
agencies and the direction was not toward any one uniform progranmatic end 
as the General Evaluation Standards published in 1975 were to require. 

Reaction to the General Evaluation StarxSards as published in the Federal 
Register was, to say the least, mixed. At this writing, some five years 
later, the original attitudes of the various state agency directors and 
administrators have not changed much! 

In an effort to aid state agencies in their evaluation building capacity, 
RSA designated somevdiat over $750,000.00 of its 1978 funds to be used to pro- 
vide contract dollars to six state VR agencies. These contracts, awarded on 
a oonpetitive basis, were intended to allow state agencies to develop "model 
program evaluation/management information units." Fully integrated into 
these "model unit" contracts were plans to field test other RSA evaluation 
areas being developed, e.g., the new Federal Evaluation Standards developed 
by Berkeley Planning Associates, and the Facility Information System deve- 
loped by Walker and Associates. The inclusion of these two field tests into 
the model unit contracts represents a significant new thrust by RSA to test 
new systems in VR agencies prior to possible nationwide inplementation. 




Ihe field tests will allow rsa to gather infornetion related to the 
validity and reliability of the data collection instrurtent proposed, the 
visefulness of the data to the state agencies, the costs, both actual dDllars 
and staff tiire required to iitplement the standards and facilities systesns, and 
the states' overall reaction to the twD proposed activities. The results of 
these field tests will allow PSA to have definitive input from state VR 
agencies on their proposed systans and standards prior to national iirplenen- 
tation. It is hoped that these pretests will provide for a nore useful and 
palatable product at both the Federal and state levels. 

Ihe model units' pretest activities of the new standards and the Facility 
Inforrnation System coirprise only part of the overall evaluation system being 
developed at RSA. In addition, the RSA Management Information System (MIS) 
is undergDing developirent throu^ a contract with Abt Associates, Inc. In 
an effort to allow for state agency involvement and inpxit into this develop- 
mental effort, the six model unit states, as well as other states, have been 
invited to review and oonment on the MIS as it is being developed by Abt 
Associates. 

Still another evaluation effort being promulgated by RSA is the states' 
use of the San Diego Case Review Schedule (CPS) developed by the Regional 
Rehabilitation (Continuing Education Program (RRCEP) at San Diego State 
University. Ihis instrument assesses compliance of casework activity with 
Federal regulations and guidelines. RSA has provided a long-term training 
grant to the San Diego RRCEP to provide every VR agency training in the use 
of the CRS over the next three to five years. 

These four efforts (new standards, FIS, the RSA- MIS and the CRS) appear 
to reflect an increasing concern by RSA to move forward with systans and 
efforts vMch will allow for more uniform evaluation of VR programs 
nationally. The model unit contracts encourage the development of new eval- 
uation capacity vMch can be generalized to other state agencies. 

Collectively the six MU states are involved in nutverous areas of 
developmental work. Ihese areas include similar bemfits model system eval- 
uation, needs assessnent for independent living programs, nanagement para- 
digms, new evaluation use of data systems, installation and use of word 
processing equipment in evaluation, case closure (case analysis) systems, 
management information needs assessment, client follow-i:p approaches and neny 
more. 

One of the major thrusts of the MU contracts is in the area of 
dissemination and utilization of contract activities. A concentrated effort 
is planned by the six MUs to make this information on their activities avail- 
able to other VR agencies. These efforts will include a newsletter of vdiat 
is "going on" in the six states, a special "manuscript" publication vAiich will 
cover specific contract products in scxcb detail and at least one national 
conference, probably in the early fall of 1981, to give all VR agencies a 
chance to "shop" among the six states and examine the products they have 
developed. 
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It appears to vs that RSA is attaipting to provide "seed ironey" to the 
six MU states. With those funds and the products developed all other VR 
agKKJies will have available to them new evaluation tools, methods or systems 
to which they mi^t not otherwise have access. Ihe MU state agencies have 
the responsibility with RSA to share and, v*iere possible, provide technical 
assistance to states v*iich wish to pursue the possible iirplementation of any 
product developed as a part of the MU contracts. 

The activities mentioned here are all a part of a new PecSeral interest 
and effort to raDve program eveiluation in VR into a new era, albeit these 
efforts in and of themselves will not be enou^. Ihey do, however, represent 
a significant new "push" to respond to state and Federal legislators' desire 
for accountability. 

Program evaluation's future directions can be sxiiroed up as "...one 
reasonable way to close the g^ between program decision making, program exe- 
cution and program revision." (Brown and Pethtel, 1974, p. 318) For this 
to happen however, there are four requisites: 

1. Program evaluation most beocxne a function of n^nagement. 

2. Evaluatoirs must be aware of and function within the politiccLL 



3. Program evaluators must accept that there is a a...e of skills 
(management oriented and methodological) necessary to fulfill 
the responsibilities of the position. 

4. RSA must continue developnnental efforts in the area of program 
evaluation. 

Finally, Peter Drucker, in his ooiun e nts on public administration, states, 
"Cne can only learn by feedback, and we know that feedback from results always 
improves perfomance capacity and effectiveness." (1980, p. 104) Program 
evaluation can be the mechanism that provides feedback to inprove perfomance 
capacity and effectiveness. Uie feetSback will be in terms of counselor and 
caseload performance, ways to present information in a timely fashion for use 
by decision makers (standards, FIS, RSA, MIS and CRS) , benefit-cost analyses, 
better ways to integrate resources, client long term follcw-i:p and follow- 
along, and the assessment of other agency outputs such as staff vork and other 
si^jport services. 
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TOE DURABILITY OF A MCX}EL EVALUATION WIT 



WILLIAM W. JENKINS, ED. D. 
PENNSYLVANIA BUREAU OF VDCATICNAL REHABILITATION 



The specifications for this contract required the developmant of a 
McxJel" Program Evaluation and Managenent Information Svpport Oait. In this 
context the term "model" would appear to have several levels of meaning. A 
model might be considered to be an ideal of exenplary method of functioning, 
■nils was clearly one aspect of the Rehabilitation Service Administration's 
(RSA) intentions since the six state units were to serve as exaRiples that 
given sufficient resources. State Agencies, regardless of size, can develop 
?Q^^^^^^ evaluation cajacity" (U.S. Office of Hanan Develoiitient Services, 
1978) . However, a mDdel also has a theoretical or scientific level of 
neaning vMch is to serve as "a representation of the underlying structure 
of a process or system. The system might be conceptual, ideal, or real. In 
general, a model has a sinple and/or manipulatable structure relative to the 
systan it represents. By making explicit the inplications of alternative 
assurrptions regarding key relationships of the issue or system under stixiy, 
a model can provide a clearer understanding of these relationships" (U.S. 
General Accounting Office, 1978) . we now recognize mDdel construction and 
nodel description as an essential portion of tte contract's vork. The 
Request for Proposal (RFP) contributed to the process of mDdel articulation 
by specifically requiring a continuing series of deliverables on the concep- 
tual framework, operational tracking system, and evaliaation plan. In doing 
so, it provided both a focus for many of the conceptual activities and 
parameters for them. 

These activities, at first glance, would appear to cOTprise a series of 
logical and rational functions vrfiich could be conducted at a relatively 
abstract level. However, in an organization which already had an established 
Program Evaluation Unit with its own identity and history, such actions in- 
volved considerably more than we anticipated. The process of reoonceptualizing 
the goals and nature of the existing unit took us into an extrenely sensitive 
and difficult process. In psychological terms it required the development of 
the self-conscious to a much greater degree than had previously existed. In 
looking at the objectives, structure, organizational relationship, activities 
conducted, research strategies and the like, we began to encounter both 
personal and structiaral resistance that needed to be worked through. Ttiis 
process was both demanding and tiite-oonsuming. MDst of the first year of 
contract operations was spent in conceptualizing, plannir^, and organizing 
the Model Unit and its initial work. Since further deliverables are 
required in these areas throughout the contract, a significant portion of the 
second year and, we suspect, third year will also be consuned in resolving 
our "identity crisis". Even after eighteen itonths of operation, the resolu- 
tion is not yet clear to us. Unfortunately, the pressure of other activities 
in the contract and continuing revisions in scheduling and work requirenents 
has reduced the time available for reflection and analysis of the irodel. 
Despite these limitations, the opportunity to reconceptualize the role of 
evaluators and information in vocational rehabilitation (VR) agency has led 
to significant alterations in our previous viewpoints as well as served to 
stimulate staff professional development. 



In developing a MDdel Unit, one of our prel'iminary ooncems vas to 
structure this unit in such a irenner that it would have the greatest possible 
iirpact on the agency's policy making, planning, and program operations. Ihe 
RFP required that the unit look particularly at the "direct linkages" it had 
with the sections responsible for each of these functions within the agency. 
In attempting to situate the mcxJel imit within the agency's organization 
and to develop a "model" structure, it seemed inportant to consider the 
characteristics of the state agency in terms of its organizational structure 
and hierarchy. The Pennsylvania agency was awarded this contract in the 
category of a large agency (Section 110 fund over 25 million dollars) and, 
with its size, the agency certainly possesses the characteristics of a large 
bureaucracy. The literature on organizational theory and strxxrture (Porter, 
et al, 1975) suggests that a large organization attempts to maintain its 
existing state of equilibriun. Although we vgould arg\ae .that the Pennsylvania 
agency is remarkably fluid and dynamic for its size, it must necessarily 
possess some degree of this characteristic. Even though change could have 
been initiated in the organization through revolutionary methods, the exist- 
ing Program Evaluation Unit as well as the prevailing itanagement philosofiiy 
showed a clear preference for an evolutionary style of iirpact. This logically 
requires that the sources initiating change be present over some significant 
degree of time. Therefore, durability was considered to be a key element in 
establishing this unit and structuring its functions so they would continue 
after the contract period. 

In reviewing the RFP, it was clear that the Federal government intended 
for the unit to continue after the contract, and in fact required a oamdt- 
ment fron the agency that it would do so. Despite this condition to the 
contract award, as a resuj-t of economic changes that have occurred recently, 
there now appears tr ' e sore threat to the continuation of Program Evaluation 
both at the state and national levels. As Taylor (1979) ir>dicated in his 
brief review of the development of Program Evaluation, this area is rela- 
latively new for rehabilitation agencies. Although the l!)€6 amendnents to 
the Rehabilitation Act are probably the first precursors of program evalu- 
ation in vocational rehabilitation, the 1973 amendments clearly established 
this function within most agencies. However, problems with the Federal 
Program Standards, the ambiguity of mandating legislation as well as the 
defensiveness of nany managers across the country has led to continuing 
resistence to its operation. The latest legislation no longer specifically 
identifies program evaluation as a required conpDnent of VR operations and 
leaves its place ambiguous. When oontoined with the austerity of low state 
budgets at tKis time and the forecast for even greater cutbacks in the future, 
the utility and cost-effectiveness of program evaluation within state agencies 
is being closely scrutinized. Duration or even survivability becomes a ger- 
mane issue for all Program Evaluation and is especially relevant for a Model 
Unit. However, this crisis also offers exciting opportimities for the model 
to clearly prove their utility and to demonstrate an ability to inprove 
operations and make them more efficient in such pivotal times. 

At the inception of the contract, Pennsylvania had a Program Evaluation 
Section which had evolved several years earlier frcxn the statistical and the 
research units. The section consisted of 8^ full-and part-time professional 
and clerical staff with an additional four menbers in regional offices vAio 
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served as Case Evaluators for field studies. Both the Evaluation and Manage- 
ment Infonnation Sujjport functions of the agency vgere conducted vd.thin the 
section, although data processing vas handled by a different vinit. The 
section vas supported by an IBM 370/158 oarputer through on-liiie terminals 

all field offices and central office, vd.th the section given high priority 
for use of these facilities. However, there was not any capability within 
this system for high level statistical analysis through a package conputer- 
ized system. In the organizationa hierarchy, Program Evaluation Section was 
renoved two levels from the agency's director. 

* J^y^ beginniiig it was felt that the model unit's inpact and chances 
of Its voric being durable would be substantially increased if it could be 
closely tied to the existing Program Evaluation Section. By doing this, the 
existing section's reputation, linkages, technical facilities and other 
resources could be utilized from the onset without the nodel unit having to 
do all the original developmental work. Oherefore, the MxJel Evaluation Unit 
ras made a separate unit within the existing Program Evaluation Section. The 
Ajal structure has the advantages of providing centralized information and 
budget control while giving special enphasis to the contract activities. - 
IJie dual structure also provides research and field support at an indepth 
level while maintaining flexibility and the experinental quality of the unit 
without formally ooranitting it to a particular organizational structure. 
Under this system the Principal Investigator is responsible for conceptuali- 
zation, research and technical direction as well as coordinating staff efforts 
on activities. The Project Manager is also the adninistrator of the Evalu- 
ation Unit and provides a<ininistrative and budget direction. Bie manager also 
IS respoisible for coordination between the unit and regular section and the 
agency. 

The REP required that the unit be situated organizationally so that it 
IS directly responsible and reports directly to the agency director, v*o is 
also the project director. This innovative and essential feature has been 
particularly important in facilitating direct and frequent coimiunication 
betjgeen the Model Unit and the agency's higher organizational levels, and 
has provided additional authority to the unit's staff during the first year. 
This has also provided valuable insight into the policy and managanent 
decision-making process that would not have been attained otherwise. 
Additionally, a coordinating oGrmittee oarposed of central office staff and 
administrators was organized to provide ii^jut and to assist in planning and 
coordinating efforts of the unit across departmental lines. Overall, the 
placanent of the Model Uni t within the agency and the developnent of its 
linkages with other sections has been designed to both increase its effective- 
ness and to structurally integrate it into the existing organization in the 
most conducive manner. Throughout the contract, resources, consultation and 
training opportunities have been shared between the agency's regxiLar and 
model evaluation units with the deliberate intention of gradually merging 
their skills, activities and functions so that units may have a consolidated, 
single identity by the contract's end. 
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Several other factors were designed to insure the lanit's iitpact and 
durability. If a unit is to be accepted within an agisting organization it 
shoiild iitined lately begin to produce work that is of iirpDrtance to the agency' 
recognized ooncems and needs. In this instance, several projects were 
selected that required field studies, and the unit's case service evaluators 
were ixtmediately put to work on them so that products were available within 
several itonths after the contract's initiation. In addition, selecting a 
xanit staff experienced both in evalxiation techniques and agency operations 
vas also iimensely useful in moving rapidly to an operational capacity. As 
Patton et al (1978) noted in his discussion of the utilization of evaltiation 
research, tSe "personal factor" which includes elenients such as interest, 
determination, leadership, and camdtment determines vdiether evcduations 
have an iirpact. Such factors were given careful consideration in selecting 
staff for a model operation and are now clearly recognized as iirportant. 

Our esq^eriences to date in constructing a "model" evaluation unit 
suggest that this process is considerably more difficult and time consvming 
than had been previously thou^t. In particular, the need to focus on the 
design and structural characteristics of the urit, both to increase its 
durability and to esqperimentally find the best organizational structure, 
requires more conceptualization, organizational si^aport, and participant 
ccmnitjnent and flexibility than was anticipated. The payoff to both 
participants and the organization, however, also appears significantly 
greater than imagined. The long-term issue of durability and continuing 
iitpact appears to be a necessary consideration in the original design of a 
"Model" evaluation unit if it, in fact, is to endure. 
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ESTABKESHING A PROGRAM EVALUATICN IMIT WITH M.V.R.B. 
(MISSISSIPPI VDCanONAL REHABILITATICN FOR THE BUND) 



Eoononic conditions are forcing both the Federal and state governments 
to nore closely scrutinize the funding of service programs such as VR, 
With emphasis being placed on cost-effective spending in the delivery of 
service, it becomes e^qpedient for the VR agencies to take a closer look at 
themselves and to particularly focus their attention on the role of program 
evaluation within the context of their organizatioiial structures. For soms 
VR agencies, blind or general, this closer look will reveal the need to re- 
assess and redefine the scope and duties of their existing program evalu- 
ation component; for others, it will mean the creation of a program * 
evaluation oonponent virtually from scratch; for all, it will corns to mean 
the utilization of program evaluation to actualize what program evaluation 
should acooitplish — 

a) an objective assessment of what is happening in the total agency 
be it good, bad or indifferent; 

b) determing why the good, bad or indifferent is happening; and, 

c) offering to agency management alternative approaches, system 
creation or modification, new or modified system linkages, and 
planning strategies to enhance the good; eliminate the bad; and, 
destroy the indifferent. 

The ultiiTBte good a program evaluation oonponent can do for an agency 
through sound evalxaative practices, is to iitpact irenagemsnt at its various 
decision-making levels so that the overall agency mission of rehabilitating 
clients is done efficiently and cost-effectively. It is rot unrealistic to 
project that funding for VR programs will be determined on the basis of 
efficiency and cost-effectiveness within the next few years. It is there- 
fore timely for VR agencies to move past the counting and reporting of numbers, 
token program evalxaation, to irKiepth, agency-wide assessments of what is being 
achieved or not achieved. 

Until Mississippi Vocational Rehabilitation for the Blind received RSA 
Contract 105-78-4005 to create a Model Evaluation Unit within the agency's 
structure, program evaluation activities constituted but one of the various 
duties assigned to the Agency's Adninistrator for Program Svpport and his 
one staff person, a Program Envelopment Specialist. While MVFB has a long 
history of excellence in rehabilitating blind clients, program evaluation was, 
at best, a nunbers game. In submitting the proposal for the aforementioned 
RSA contract, agency management acknowledged that vMle it recognized the 
need for structured, indept±i program evaluation, the agency sinply lacked 
funds for establishing a coiponent of qualified persons to be solely concerned 
with evaluation activities. 
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The purpose of this article is not only to call VR administrators' 
attention to the fact that program evaluation shDuld be handled by a ocmponent 
or unit solely, or at least priitarily, concerned with evaluation activities, 
It IS also to share with other blind and general VR agencies how MVRB got its 
unit on Its feet and running once funds were made available. Hopefully, our 
experiences will assist VR agencies in \?)grading their existing program eval- 
uation carponents; assist in the establishnnent of units where none exist; 
and, encourage all VR agencies to mate program evalmtion meaningful enough 
to reap maxiimm benefits from the tijte and effort spent. 

PRIORITY NIMBER ONE 



In order for evaluation as a practice to produce anything worthvAiile, 
and in order for a program evaluation unit to iitpact an agency's operation, 
agency management from top-line to seoond-third-/and fourth-line administra- 
tors or s\?)ervi9ors must realize that they have a vested interest in the 
evaluation process. Evaluation is a term that either evokes total apathy or 
sudden panic. It has somehow core to connote "blame" as oppDsed to the 
objective reporting of factual information. This obvious problem with seman- 
tics may well explain v^y toten evaluation has been so raitpant in VR agencies. 
Evaluation is nothing more than objective assesanent of an activity. If an 
activity has been well planned and conducted according to plan, an evalx^ation 
of It can only reveal that v*dch happened as a result of conducting ths 
activity. Assessment of the activity as a ptese of the evaluative process can 
only reveal whether or not the activity helped reach a planned goal. In 
essence, evaluation does not take place for the sake of fiiding fault or 
placing blame; is should take place for the sake of effectiveness and effici- 
ency in meeting goals. The managanent of an agency itust then clearly have a 
vested interest in the evaluation process v^ch mist include (1) a coimitment 
to contribute to the process by sound planning viiich lends itself to sound 
evaluation; (2) a comiitment to test, if applicable, evaluation outcomes; and, 
(3) a ccmndtment to utilize evaluation outcomes to increase individual, com- 
ponent, and agency goal attairment. 

IXDGISTICS OF GCTTING THE MVRB UNIT STARTED 

As is the case with MVRB, it may ironically turn out that funding will 
come to be regarded as one of the least of problOTs in establishing a program 
evaluation unit. 



Dr. John Gehi, Director of Institutional Research at Atlanta's Spelman 
Cbllege, sent our MEU a copy of John Gall's book, in* viiich Gall states that 
"People In Systems Do Not Do Wliat The System Says They Are Doing," and further 
notes that Murphy's Law, "If anything can go wrong, it will," is still valid. 
While It can be reasonably concluded that Gall did not have the logistics of 
establishing the Mississippi MEU in mind \*en he authored SYSTEMANTICS, it 
iTust be conceded that establishing the Mississippi MEU coiald have served as 
a model for his book. 



Because our fundings was 100% federal noney for an experimental project 
of three-years duration, some unique problems had to be solved in < jrder to 
hire st2rff and e9q)end funds. The first problem with hiring staff had to do 
with MVFB meeting the requirements of the State Classification Oomttdssion. 
This Oonmission designates the nunber of job slots and job position titles 
a state agency can have. Additionally, this OcOTidssion determines the re- 
quirements a person must meet to be hired in a specific job position. Once 
a state agency's job slots and position titles are set, it is a rather 
lengthy process involving both the state agency and the Ctonmission to add 
staff to the agency's list and to add new titles. 

Since the MEU had to initiate and coirplete its work in accord with the * 
mandates of the RSA oontiract, the least time oonsuoujig solution to the problem 
or hiring staiff was to siitply designate, on paper, the MEU as a state agency 
within a state agency. Ihis sounds peculiar and does require further ejq>lana- 
tion (Gall was right!) . As a state agerxyr the MEU oou3d get approval of job 
slots much fester than approval oould be secured for MVRB to add to its exist- 
ing slot allocation. Thios, the C3reation of the MEU as a state agency within 
MVi© allowed it to meet its contract mandates and at the same time satisfy the 
State Classification Conmission's requirements that (1) at the end of the 
three-year period the state agency status of the MEU woiild cease along with 
Federal funding sx^jporting it; (2) cdl staff hired by the MEU would be hired 
in accordance with the rules and regulations of the Classification OOTirdssion 
requirements for similarly existing jc±) position titles. In the RSA contract 
and systan, the Unit's Technical Writer is called a Technical Writer; however, 
in the Classification Qximission's system, the Technical Writer is called a 
Public Information/Relations Writer II. 

The matter of hiring staff involved yet another hurdle; that of getting 
approval to e:5)end funds provided by the contract. J^roval for state agencies 
to e)5)end funds and the procedure by vAiich funds can be e3q)ended are regulated 
by the State Budget Ocxrmission. Since the MEU's Federal funds were subjected 
to state regulation, aj^roval from the Budget Oomnrassion was needed for such 
things as purchasing, leasing, renting of equipment, salaries, and for all 
purchases requiring oonpetitive bidding. 

The Capitol Gonmission of the State of Mississippi regulates and approves 
the square footage of physical space that can be used by state agencies. Tliis 
square footage for the MEU was determined on the basis of the nunber of enploy- 
ees to be 0iysically housed. Ihe MEU's Principal Investigator had to first 
locate suitable office space basoi on Capital Oorannission requirements and then 
appear before the Capital Oorannission to explain (1) why MVFB needed additional 
space to house the MEU: (2) hew the additional space would be paid for; (3) vrtiat 
the MEU was designed to accoitplish; (4) v^ere the proposed space was located; 
(5) how long the MEU would need the space, and so forth. C^ice approval was 
given by the Capitol Qxmdssion, the MEU moved into its offices. 

It is likely that most VR agencies have similar oormissions or boards 
that regulate ertployroent, housing, and budgetary matters. Therefore, the 
approval or lack of approval from svch boards and comtassions has a great 
bearing on getting a program evaluation component started. Based on our 
esqperiences, we would reconmend that a VR agency do the following prior to 
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ejq^ecting the program evaluation to begin functioning: 

1) Make a siirple listing of all board and/or corandssions v*K)se 
approval will be required to establish the program evaluation 
ocatponent and use the listing as a checklist to make sure that 
all needed sarictions are secured; 

2) If it appears that the sources of funding or scms other exogenous 
factors may hinder irtplementation of the oorrponent, make a list 
of alternatives solutions to sutrtdt to regulating boards and/or 
ocHTiviiss ions ; 

3) If funding for the program evaluation unit is to be a included 
in the agency's annual budget, prepare such justification as 
may be necessary to sv^^ort the request for additional funds, 
transfer of funds, etc. 



TOE CRUCD^Ii MfiJTER OF STAFFING TOE UNIT 

The staff for a program evaluation unit will depend on (1) vrfiat the 
agency expects the unit to acoorplish, (2) the size of the agency, and 
(3) funds available for hiring evaluation staff. Hie MVFB MEU consists of 
four staff person: a Principal Investigator who directs all unit activities 
and has final determination of unit projects; a Systote Analyst v*k) is 
responsible for the developmsnt and modification fo agen::y systems, for 
the design of arxi modification of agency systons, for data interpretation 
and for the design of data collection instrurnsnts; a Itechnical Writer 
vjbo is responsible for formatting and writing all MEU reports and/or docu- 
ments and for assisting in evaluation activities conducted by the MEU's 
clerical, secretarial, and general office nenagement needs. Ihe MEU serves 
an agency that ha? 109 job position slots. When the MEU contract expires, 
its work will be jdiased into the MVIB Program Support Unit vrtiich will assure 
full responsibility for all agency evaluation activities. 

One valid method for determing vdiat type of staff will be needed for an 
evaluation unit is to simply look at v*iat is required of the agency and use 
those requirements to determine vtet kind of staff persons vgould be mDst 
appropriate. The MVPB unit, in the process of developing a Basic Management 
Paradigm for the agericy, developed a natrix of twenty (20) oonpliance act- 
ivities for which MVFB is responsible during the course of a fiscal year. 
These conpliance activities run the gamut from routine reporting to continued 
studies to researching, writing, and iraintaining on file cartain reports 
which are open to public scrutiny. Using such a matrix, an agency could 
well staff its program evaluation unit on the basis of vAiat type of people 
are needed to insure agency oonplaince in various areas. Since corpliance 
activities should be one of the major areas evaluated annually, a jprogram 
evaluation unit could assist agency management in developing a flaqging 
system to insure that corpliance activities are taken care of on a timely 
basis. 
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In developing a new program evaluation unit emd in upgrading exisiting 
units, VR agencies should (1) determine, in writing, viiat they are respons- 
bile for internally and externally, (2) plan activities that lend thanselves 
to evalxjation, and (3) select staff vdio will not only conduct evaluation 
activities to detanine if item #1 is haH)ening, but v*io can also contribute 
to the agency's planning of activities, initiation of activities, cind revi- 
sion of nodification of agency activities. At a ininiinLim, the Mississippi 
MEU recomnend that a program evaluation unit consist of: 

1) an adninistrator thoroughly familiar with the agency's operations, 
external requiranents (federal cind state) placed on the agency, 
cind the state-of-the-art in evaluation; 

2) a systems ar^yst capable of developing or modifying agency systans, 
capable of determining methods of using da.ta processing within the 
agency, and capable of contributing to other facets of the evalua- 
tion unit's duties (i.e. writing, designing data collection 
instnments; 

3) a writer capable of preparing evaluation outcomes so ttat they 

are understood by agency staff, capable of assisting in the pr^>ara- 
tion of agency training and/or operative manuals, cind capable of 
assisting in cind contributing to the other facets of the program 
evaluation unit's duties; 

4) a researcher/statistician capable of iranipulating statistical data 
as needed in r^30rts, documents, etc., capable of researching gen- 
ereil subjects for information needed by agency oarponent and itamge- 
ment and capable of contributing to other duties of the program 
evaluation unit; cind 

5) a secretary, who, in addition to handling the unit's secretarial needs, 
can also contribute to the unit's functions throxigh such skills as 
establishing and managing a unit library, establishing and itanaging 

a central file for unit products, agency reports for public use, etc. 

While each staff person in the Mississippi MEU has specific duties to 
perform, each person is also involved in the planning, researching, and 
reporting activities of the unit. For exaitplo., in addition to his adminis- 
trative duties, the Principal Investigator also does report writing and edit- 
ing, research, oral presentations of evaluation outcones, and statistical data 
analyses to name a fev. The MEU Secretary, in addition to her normal duties, 
is currently corpilijig data being collected in a phase of the pretest of the 
proposed Standards developed by Berkeley Planning Associates, Inc. Ihus, a 
good program evaluation team should be a itulti-skilled group of persons cap- 
able of performing beyond assigned duties. 

INTR(X)UCING TOE PROGRAM EVZ\LUATION imT TO TOE AGENCY 

Earlier in this article it was noted tViat VR agency administrators laist 
ooiTB to see that they have a vested interest in program evaluation in ordar 
for evaluation outccines to be of any significance. All agency staff nust 
likewise cotia to see evaluation of their individual cind team efforts as a 
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step toward inproving overall agency perfonrence, not just performance in 
their own private sectors. Very cognizant of the coitmDn connotations of 
the tern evaluation", the MEU tried to get off to a good start by explaining 
i?™^ to all MVRB staff. This was dDne tIiroi*gh using established vehicles 
of oortimmication in the agency (i.e. Regional Directors MeeUngs, Agency 
Newsletter) to convey in general terms haw evaluaUon functions inln ageix^y 
and how individuals and conponents could use the services of the MEU to 
better plan and meet individual, component, and overall agency goals. 
Additionally, the MEU visited all facilities which provide services to MVRB 
Clients, participated in joint projects with the MVTB Program Support Unit. 

■ °'^v.^'^.''^'^}.^'^^ ^^^^ P^^"^ ^ infontally discuss their needs 

and how the MEU could assist, and began publication of a single-sheet mDnthly 
Xeroxed newsletter to keep all agency staff abreast of what the MEU was doing 
and planning to dD. ^ 

MPii .Sli!.;?^^/^ "getang to know each other" paved the way for further 
^ffiU and MVRB staff ooitinunication. Thus, the MEU was justifiably pleased 
'^i^^-S'-^^f ^ ^^^^'f responded positively to participation in data collecUon 
for the BPA proposed Standards. Even though participation by the District 
Offices ineant an increased work load, no unwillingness to participate was 
exhibited. This action, we feel, underscores the value of properly intro- 
ducing evaluaton as a concept and the evaluation unit to an agency. 

Since much the evaluation that takes place in VR agencies is directed 
at the counselor-ievel, the MEU imnediately looked for a iteans of establishing 
and maintaining cortitunicaaons with this level. Beyond informal interviews 
and including a "book review" section which reflects the state-of-the-art 
in counseling in our monthly newsletter, the MEU also iniUated preparation 
of monthly graphs to illustrate to counselors exactly where they werTin 
terms of reaching numerical goals set for reliabilitaUon of clients. Each 
counselor received his/her individual "Tine and Milestone Chart" which was 
not distributed to other agency staff. The counselors supervisors received 
a simlar chart reflecting the district's progress and the Director of Field 
Services received a copy of the district chart and a statewide cumulative 
chart. These charts were a clear statenent from the MEU to counselors that 
their progress, or lack of progress, toward a monthly goal was infontation 
charted exclusively for their constructive use and informtion which was 
being judiciously handled by the MEU. Of course, a counselors progress/lack 
ot progress toward a numerical goal was not a secret undiscoverable by the 
supervisors, but the MEU opted to handle the charts it produced for counselors 
(who are often "blamed" for an agency "lack" of success) in a manner to in- 
spire confidence and hopefully to eliminate the negativism associated with 
evaluation products. 

Since the methods of introducing the MEU to MVFB staff were carefully 
thought out and have thus far served a good purpose, we suggest that other 
VR agencies utilize any of them that they deem workable in their agencies. 

WHAT DOES THE PROGRAM EVALUATION UNIT DO ? 

The Mississippi MEU has a number of activities it must perform as 
mandated by the RSA contract funding it. These sane activities are also 
required of the other five VR agencies vMch received contracts. In addition 



to fulfilling contract nandates, the Mississippi MEU has conducted routine 
and special evaluation projects for MVRB. A partial Ustinq of these act- 
ivities include: "«=oc a^u 

1) An analysis and subsequent revision of the MVRB form and procedure 
used in conducting the annual Review of aients in Extended Eltplov- 
nvent; ^ ^ 



An analysis of all forms (state and federal) used by MVRB with 
subsequent reconmendations to agency management based on findings; 

Developnent of a prcjcedure for agency response to external requests 
tor information (special reporting) ; 

^^aration of the agency's annual Quality Cbntrol/Case Review 
iteport in conjunction with the MVRB Program Si^jport Uhit; 

Deyelopnnent of the facility reporting forms used in the Facilities 
Utilization Reporting System developed by the agency and RSA per- 
sonnel based in Atlanta; 

Development of the proposal and securing of partial funding for 
MVRB data processing equipment; 

Oonstruction of an Qiployer Assessment Questionnp.xre to be used 
to collect follow-up data on MVRB clients plaired into the ccmpe- 
titive labor force by MVRB Affirmative Action Sptx;ialists; and, 

Developnent of a Basic Management Paradigm vAiich is not yet 
ocmplete but has included (1) an analysis of MVRB staff positions, 
and . (2) the design of a matrix of MVRB ooipliance activities. 

u M,S®^-^^' listing of MEU activities reflects that program evaluation 

at MVRB IS responsive to specifically identified needs and encottpasses the 
total agency. 

v_ 1?^^"?^^^.^ ^ agency might use to determine what program evaluation 
snould address is to develop and prioritize a oanpliance matrix so that the 
unit can proceed step by step. Another approach that would be workable is 
tor an agency to determine its areas of strengths and weaknesses and let the 
program evaluation unit begin by addressing these areas. Still another 
method is to let a program evaluation unit begin by addressing one major 
facet of the agency (i.e. a particular ccnponent, the decision-making pro- 
cess, the process by vrtiich agency goals are fontulated) . 

Evaluation is not, nor should it be considered, a panacea for all of an 
agency s ills. Yet, by determining a workable a^aroach, a program evaluation 
unit can key m on specific ills and work its way around to offering renedies 
for same. Eventually, the total agency can be addressed. 



2) 
3) 
4) 
5) 

6) 
7) 

8) 
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IS A PROGRAM EV?ajJATION IHET WDREH THE HASSLE? 

Perhaps at the end of our three-year contract, we can provide a profound 
and philospEAiical answer to this question. As things stand now, our best 
answer is a siitple "yes", establishing or upgrading a program evaluation imit 
is well worth the hassle. Ihe need for sound evaluation in VR is not going 
to diminish, it is rapidly beocming required. The uses of program evaliaation 
outocnes and the areas vAiich p rogr am evaluation can address are limited only 
by how program evaluation is conceived by agency itanagenent. Sound program 
evaluation conducted in an agency genuinely concemed vdth efficiently and 
effectively delivering services to clients will yield valid data for decision- 
making and thereby iitpact the agency's mission. Conversely, token evaluation 
will yield token results. Establishing or upgrading the program evaluation 
vnxt in VR is well worth the hassle. 
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MftNftGTNG REQUESTS FOR EVALUmCN INroRMAnCN 

LAUPIIEE B. HATCHER 
STATE OF OREGCN DEPARTMENT OF HUMAN RES0UR3S 

4.K« Evaluation infonnation is needed for decisionitaking at every level of 

administrators, planners, program nenagers a«a counselors 
resources allocation, accountability, policy formulation, 
policy evaluation or utproving programs, they need evaluation infonnation. 
^ni"=L?fi agency this need can soon exceed the capacity of program eval- 
V^S;^^^ . ^"^"^^ information. Iflhen sua. a need beccites excessive, the 
information retrieval process can beooite very frustrating; without a struc- 
tur^ ratiOTal process to manage requests for information the system may 
become frau^t with inefficiency, missed schedules, dupUcation of woric and 
^if?J-^^^^^?*^- . '^^ following pages describe how the Oregon Vocational 
Rehabilitation Division uses a service request process to nanage information 
resources . 

f^^v^^^!^°f^4°\''^^'^'^ ^ managed jointly hy the Autcmated Data Processing 
(Ara> Unit and the Program Evaluation and Statistics (PE«<S) Unit, Oregon's 
model evaluation unit. These are tro separate administrative units with the 
managers of both units responsible to the Deputy Administrator. Hie necessary 
link between the two units to permit PE«S to manage information is achieved 
through the service request process and a PESS researr:h analyst assigned as 



The ADP Unit oonsists of a imit manager, five systems analysts and 
prograiiitiers, and three data entry and production staff. Hiey have a 
remote job entry facility which permits submitting jobs to a coitputer housed 
several miles away in the Department of Human Resources cortputer center. 
Hie data processing staff has responsibility for maintaining and refining a 
number of systems, including the Client Information Teleprocessing Systan 
(CI'IPS) , the acoounting system, payroll and accounting for the state-operated 
facility, and case files and acoounting systems for the Disability Deter- 
mination unit. In addition to these systems, the ADP Unit maintains and 
refines routine management information reports and responds to special 
requests for information. All special requests for information from 
existing files, requests to add or delete data elemants in an automated 
file, requests to create or discontinue an automated file, and requests to 
create, discontinue, or modify a routine management information report are 
docunented as service requests in the PE«S Unit. 

The PESS Unit oonsists of the imit manager, a managoiEnt analyst, three 
research analysts and a clerical specialist. This unit has responsibility for 
forms management, purchasing of printing and copying services, maintenance of 
the agency policy and procedure manual, special research projects, program 
evaluation and follow-up studies. Federal and state reports, coordination of 
the service request process, and information retrieval. Members of the unit 
are skilled in the use of the Statistical Package for the Social Sciences (SPSS) 
and Easytrieve, a basic retrieval language, and use these skills to meet special 
information needs. The various members of the unit have been assigned as 
specialists in the different activities and serve as liaison with various man- 
agement imits. This liaison assignment permits development of rapport with 
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prog ra m managers as well as development of e>qDertise in specialized subject 
areas. Hiis facilitates analysis of problems and the proposed solutions. 
When any agency staff heis a need for information, they discuss their need with 
a PE&S Unit monber to begin the service request process. 

When PE&S analysts are confronted with infomation needs, they conduct 
analyses of the needs. In consultaticai with the requester, the PES-300, 
AEPA^&S Service Request Pom, is ccmpleted. (See Attachment 1.) In this 
initial review, the analysts must consider v*iether the need has been stated 
clearly. If it has not, they must continue the discussion mtil it has 
been. Ihe analysts must seek to identify v*iat data nay meet this information 
need. If the data are not available, they most consider the possibility of 
adding new data elennents to the system to provide for future needs. If the 
data are available, they most consider the various alternatives of locating 
it in existing reports, lasing PE&S staff to retrieve it from computer files, 
or requesting ADP services. If it is contained in existing reports, they 
must consider the other workload aiyi priorities of the unit to determine 
whether to direct the requester to the c^Jpropriate files or do it for him. 
If the PE&S analyst is unsure of the availability of the information or of 
the feasibility of MB? oatpleting the requests, the PE&S/AEP liaison 
analyst is oonsiiLted. 

When staff initiate requests, they must specify priority. We use a 
siirple priority schedule consisting of four rankings. A ranking of four 
means the information is useful, but not essential to the requester; a 
ranking of three means the infonration is essential to the user, as it 
results in loss of effectiveness or efficiency if not provided by the date 
needed; a ranking of two means the information is essential for the acoan- 
plishmBnt of major agency objectives and, therefore, must be provided by 
the date specified; and a priority of one means the information is considered 
so critical that the agency adninistrator or his deputy has indicated other 
work must be set aside as needed to meet the specified due date. When the 
requester specifies a priority, the PE&S analyst assures that the priority 
is consistent with the intended use as specified in the request. 

In the analysis of the requests, the PE&S analysts must assure the 
desired results are clearly specified. If the request er is representing a 
third party (another state agency) ) , the intended user may review the speci- 
fications of content and fontat. The PE&S analysts will attenpt to estimate 
the volume of output to help the requester plan for the vse. For exairple, 
if someone requests information about reason for closure and diagnosed 
disability for closures for the last three years for each local office, the 
PE&S analyst may point out that this could result in tables containing over 
200,000 values. When presented with this analysis, the requester may 
consider another format or seme grouping structure to make the output more 
usable. Other questions considered in this analysis are presented in 
Attachment 2. 

Upon completion of the analysis of vAiat needs to be done, the PE&S 
analysts must reooranend to the Unit Manager how it should be done. Hiey 
indicate vAiether it might best be referred back to the requester, oonpleted 
by PEfiS staff, or roxited to ADP. Their reoonmendations and the oonpleted 
service request form are passed on to the PE&S Unit Manager for authorization. 




if fli rSf ""^^ ^ oonpleted because data are not available or 

^J^rJX^oi^ so ocnplex that the cost jus^fication is not evident, the 
PEg Urut Manager will refer the request back to the requester. If the 
infonnatxon is ayaUable in the agency library, or iflSS other staS n«riber 
hasthe infontBtion the PE&S Unit ^fenager will decide, based on the uni?s^ 
ft P^f-^^' «^-ther to collect th4 infonratiS o?^f 1: 

■ ^^L^-^^^ Manager authorized a service request or denies it 
i^nifri?^ ^®3uest log (see Attachment 3) . ihe ' 

original is given to the assigned PEfiS analyst or to the ADP liaison 
analyst and the copy is filed in a PE&S file for outstanding re^te. 

.Yr..J^T-^ fli^^ delivers the request to ADP, basic infomation 

tSf A^LoLf;;^T^ T ^^"^^ ^«3ue^t maintained by Ara> 

(see Attachment 4) . ihe ADP Unit Manager reviews the request, consulting 

^.^l/^L^J^^^' ^ ctetermii^ the reqSst cannot be^ 
oonpleted by the date specified, they notify the requester and proiect a 
poss^e cortpletion date, ihe requester then aS^t th?pr^S Lte 
or negotiate for a higher priority to receive it sooner. 

«,=^J!^1''^2^^ ^ "^""^^ ^th PEfiS and ADP have a tracking 

^ ^ , PE&S log contains records of all ADP and PE&S requests, while 
the ADP log contaijis records only of ADP requests. An agen^AS^kr^ 
oamattee vas established to provide guidance in prioritizing and scheduling 

^ h..^^^.^"^^^^ ^ M^qer fin£ it iiiposSbS^ 

to neet all deadlines or reach ooqpronise on rescheduling, the oonmittee may 
be cailed together to review the workload, ihe ADP log of outstanding 
requests can serve as a basis for discussion and the renegotiated schedules 
can be used to i^xiate the log. ~i^«> 

When an ADP service request is ccnpleted, the product is delivered to 
the requester who initials the request form to acknowledge receipt. A copy 
of this initialed form is routed to PE&S. ihe ADP dDcumentation of the 
request and copies of the program are filed with the original request form 
in ADP so they can be rerun if needed. Both PEfiS and ADP update their 
request logs, m the case of a PEfiS cornpleted request, the process is the 
same except that ADP does not receive a copy. 

Ihe priority ranking process has proven to be valuable in managing the 
workload in ADP. Very few priority "1" requests have been necessary. Is 
work can be scheduled to meet due dates. Requesters are encouraged to 
anticipate their needs and allcw at least two wedcs whenever possible, ihe 
ADP Unit Manager receives reports regularly to nonitor which requests are 
done, which are overdue, which are due soon, and what portion were coitpleted 
by ttoLr due dates. Probably the biggest problem with this priority schate 
has been an inclination on the part of sane users to assign hi^er priorities 
than warranted by their need, ihis misuse can be controlled, however, by the 
PE&S or AEP Manager challenging the requester based on the intended use 
that was specified and by regular meetings of the ADP users' committee. 
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A nutiDer of observations can readily be made aboxit the benefits of this 
service request process in the effort to manage information. Ihe service 
request form itself has been structui»3 to lead the requesters throu^ docu- 
mentation viiich can readily be converted to coirputer program code. As 
peoEile make requests, they become more accustomed to the structure and learn 
to express their information n ee ds more specifically. As they participate 
in the analysis process, they become more aware of the potential benefits 
of the system as well as its weaknesses so they are better able to offer 
suggestions for the developnnent of the system. Outgrowths of this increased 
awareness include suggestions made to add elorents to the client file and 
requests by program managers for training in Easytrieve prograimdng. 

In suimary, the Oregon Vocatioiial Rehabilitation Division's service 
request process is a manageable approadi to information retrieval. The link 
between the Program Evaluation and Statistics Unit and the Automated Data 
Processing Unit provides for a sharing of responsibility for meeting 
infonnation needs. As program managers and planners specify their informa- 
tion needs and assist in the evaluation of data resources available, they 
beoone more conscious of hhe process required to naintain an adequate data 
base. Both content of the data base and quality control exercised in it 
become concerns of the lasers as well as the data processors. This user- 
oriented data base management increases the likelilxod that infonration 
will be available and usable vAien it is needed. 
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Title 



Requester 



AIPEAOiMENr 1 

Department of Hianan Resources 
VOCanCNAL REHftBILriiaTICN DIVISION 

PE&S/ADP SERVICE FBQUEST 
. _ Priority 1 2 3 4 # 



Date Requested 



^ ^ HO. often? once other 

Describe Project Needed: 



Data Files: Oost (include DDO ) DDO Master 

^ual fxles Oth^ — " 

Date Period: Ftoti to 

Deliver Product to 



How will results be xised? 



Where will results be filed? 



Specifications Descriptions: 
Selection Criteria 



New prog r am 



Request Type: Change file date fc...wyjLani 

Change to program permanent Library retrieval 

Change to program temporary Other 

Rerun, update program 



Printout Sequence 



Elements/Format 
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Itotals, Counts 



PE&S Authorization Assigned to 

Delivered to ADP logged out 



ADP Acct # Assigned Date 

Proj. L eade r Initials Ooqpl. ACD 

Requester initials acknowledging delivery/ooitpletion 



Oomnents 
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ATEACHMENT 2 
GUIDELBES TO REVIEW AN ADP REQUEST 



2) 



3) 



4) 



7) 



c) What turnaround evidence of change is needed? 

^n^,^^ P"^^ ^ modification such as change in date ceriod 

b) IS the data to be comparable to any other data? 

^SSaTrSSi^s^r by changing scathing not anticipated in the 
original request, such as a change in sort order 5F"added elanents 
aj Is the change clearly identified? eianenrs. 
t>) IS the data to be ccnparable to any other data? 

a Which! elanents are wanted? aearly def i^? 

i J! specified? is format flexible? 

c ^uld an existing report with modifications provide this dafc,? 

d) Dov^wantflexibiUty to rerun with. specif ic^SjeS^ 
Program a new report to becone routine, 
a Which elements are needed? aearly defined? 
D) Is format specified? 

c) What will distribution schedule be? 
a) Is it to be ootparable to other data? 
e Will soneone document for users? Who? When? 
t; Will user training be required? Why? When? 

^dnew^fUe element(s) through additional or changed i^t and storage 

a) What input process has been established? 
b Have all parties involved in input been consulted^ 
d °^ ^tructions changes necessary? Training? 

f ST? S^f^ ?^^"^ ^ ^l^t necessary? 

f ) Will we try to retroactively collect it for other rS,rds? 

Delete an existing file elenent. 

a) What is it being lased for? 

b) Have all users been consulted? 

c) What other reports/systans may be affected? 



5) 



ti) 



I 



d) lias the irpot vehicie been eliminated? 

e) Should the file retain what it has? 



8) Change an existijng internal system of processing data. 

a) What are the current uses? 

b) Have all users been consulted? 

c) Is the change clearly def inad? 

d) Is the justification documented? 

e) What problems could it create in rraking data more carparable or less 
comparable to other data? 

9) Establish a new internal system of processing data. 

a) Gould a nodification to existing systems suffice? 

b) Is the process clearly defined? 

c) Is the justification documented? 

d) Have potential \asers all been consulted? 

e) Will the results overlap with others? 

10) Qxtbination of any of the above. 
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ERIC 



MTAOlMENr 3 



PE&S SERVICE REQUEST LOG 



TIH£ 

Target tonitor Names 
Vendor File On-Line DDO 
Special Service Oodes 
Status 20 aients 
A*iin Review Data 
WCD Billing Report 
ISO 

Vendor File 

Placaitt Svc Veniir FY79 
AFS Shelt Bnpl Subsidy 
SSDI Denied Pre-Plan $ 
Revise SSA 853 
WCD Data - HistDiy 
WCD Study 
Revised R-13 
COT Study 
Cbst Edit Change 
Service Code Change 
SSI/DI 1407 Update 
Facility l^t Changes 
1407's 

1407's - aosed Records 
Alaska Visit 
I^eoognized Svc Update 
Leave Utilization 
DDO Processing Tine 
Reg 1026 Itepeat 
Msdify Examiner Tables 
Reg Brkdwn for Flowshts 
Reg Brkdvn DVRD1026 
Cleanup Exam 74 on 1026 
List Pending Over 60 



USER 

Laurilee 
Shari 

Clarence 
MiJce W. 
Tim 

Betty N. 
SueT. 
Jack 
John S. 
Isie 
Elsie 
John J. 
Carl 
RSA 
Irene 
Ken/Sue T. 
Sue T. 

Ken/Elsie 
Art 
Ken 
Elsie 

Schlicting 
John D, 
Sue D. 
Tim 

Dick D, 
Dick D. 
Dick D. 
Dick D. 
Dick D. 
Did; D. 



mm my ME oo^PI^^ 



01/12/75 3 
06/05/78 



10/24/78 

11/01/78 

03/08/79 

03/01/79 

05/08/79 

05/30/79 

05/15/79 

07/17/79 

07/27/79 

08/06/79 

06/28/79 

07/06/79 

09/05/79 

09/20/79 

09/27/79 

10/02/79 

10/02/79 

10/04/79 

10/15/79 

10/09/79 

10/24/79 

01/23/80 

11/01/79 

11/23/79 

10/04/79 

10/04/79 

10/04/79 

10/04/79 

11/09/79 



3 
2 
2 
2 
2 
3 
2 
2 
2 
2 

2 
2 
2 
2 
2 
2 
1 
3 

2 
2 
3 
2 
2 
2 
2 
3 
3 



01/05/80 
11/02/79 

10/3V78 
01/01/79 
07/31/79 
09/01/80 
10/26/79 
06/30/79 
07/20/79 
10/20/79 
10/15/79 
08/12/79 

12/01/79 
12/15/79 
01/22/80 
10/02/79 
10/12/79 
01/20/80 
10/12/79 
1V16/79 

01/01/80 
04/01/'80 
01/10/80 
11/30/79 
12/31/79 
12/31/79 
01/18/80 
02/08/80 

i:/:i/79 



X 
IH 

m 

Vm 

X/ffi 

Vm 

IH 
IH 

VIH 
VIH 
VIH 
VIH 
VIH 
VIH 

Vm 
m 

VIH 
X 

Vrm 

VIH 
VIH 
VIH 
VIH 
VIH 
VIH 



02/01/8 
04/10/8 



05/30/7 
12/07/7! 



04/04/8C 
C0V01/8C 

04/04/8C 



04/15/80 
02/22/80 



ERIC 



i«^M«ST ASSI(» TO ^ I^U^tTr. ^ OF THE I.^ RESEARB 



TRBQOST \XmL(m REKABimriON DIVISICN OOTSTRNDING REQUESTS BY PRIORITY 

A4/80 

Sm ORIGINAL OOMPIETION 

RQST ASSICU DftTE DATE DATE A! 



mm 



TITLE NO TO PRIOR RECEIVED ASSKM) NEEDED DATE USER ACCT NO 





1475 


5R 


2 


10/04/79 




1557 




2 


01/lfi/fiO 
ux^ xu/ ou 

01/22/80 




TP EDIT CHANGE/ 


1560 




2 


BRPTATPC; 1494 












1518 


SB 


2 


02/12/80 




1550 

XJJv 




2 


01/18/80 

UX/ XU/ u u 




1552 

X J J4> 


Iu\ 


2 


01/11/80 

UX/ XX/ uu 




1549 
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ERIC PS OF 04/14/80 HERE WERE 22 OUTSTANDING REQUESTS 



BOOK REVIEW 
EVALUanOJ ; PROMISE AND PERFORMflNCE 
JOSEPH S. mOLEi, THE UFBAN DJSniUIE 

Reviewed by 

Robert D. Struthers, Ph.D., Michigan Department of Education, 
Bureau of Rehabilitation 

The demand for better evalua^on continues to qrow and hnnk«? on i-ho 
^Dect are proliferating. RSA's nodel program SlSn^^S^hS Showed 
^e^S'i^pSi' ^S^l"?^ literatSrS; the subj^?^i?Lf 
T^i^ ^uS^Lh^ftSf S?? T "^^"^ evaluation concepts, discusses 
yeijeiaj. issues such as the failure of managers to utilize evaluativp informs 

^'^^f^Z^"^^ discusSS Of evaluation SSro^^des 
S^S^^ JSlf^^^.^^ different from rehabilitation. Evalu^on: 
IT^ Performance falls sotewhat in the last category but, nonethelSs 

lias a nunber of qualities that we felt w^e iitpressive/ ^ nonetheless, 

Ur^n^tT^T^°^ strategy presented in the book has been develooed by the 
Urban Institute program evaluation staff in their work with fedSafand locll 
agencies during the past decade. Much of the text consSS jfSL^ 

Institute wjrk. The author, Joseph S. Wholey, has worked^TtoS ST 
Urb^titute and within the Federal ^vemment^l^ ^t!j ^oJS 
in program evaluation in the tepartnent of Health, Education and Welfare! 

The book, therefore, has credibility because it's based L^orW k^, r«.=.^ 
ticxng evaluators in the Federal governrriirSTb^if if ^^"^ts'^ 

S^^si:?" ^ to produce useful evaluation infontation in an efficient 
SSlSSrSs^S."""^ "^'^ organizations and with iSSSd 

^^i^ Although there has been constant growth in demand for program evalu- 

^^r^C^ri^^r^'Z^' f^^^^^ have often ^SSTtT^ss 

the mark. On one hand there has appeared to be a simplistic belief that 
almost any quantitative report would have significant evaluative meanSq 
usually, such has not been the case. Instead, "evaluation" hafo^SS^ 
added new rprting burdens, but has provided data with little meaning for 
decision makers. At the other extreme, there has been a feeling that "rSl 

SS^S^S^i"^^ ^ done unless the rigor of evaluation research, complete 
with control groups, can be imposed. "-^hfj-cic 

Wholey's book does not tell how to achieve ccmprehensive evaluation of 
oonpl^ agencies with a staff of one or t^ people, but the book illustrates 
both 1^ Its strategy (moving fran evaluability assessment to intensive 
evaluation) and by its examples, how evaluated can be conducted to be use- 
f;^; ^"^^^"^^ such as a rehabilitation agency. Instead of pix>ceed- 
ing from a theoretical definition of evaluation, Wholey launcher iirmediateiy 



into vdiat evaluators c3o with program managers when a problem is presented^ 
arxi shows how the evaluator can perform a service to iranagers before a 
decision is made to launch into more intensive evaluation. 

As stated in the surmary of the book^ Evaluation; Promise and Per- 
fomance "shows how evaluators can help government managers and policy 
makers to clarify the intent of government programs and to improve program 
efficiency^ effectiveness and responsiveness. The book presents a strategy 
through which evaluators can determine vrfiat infonnation is likely to be 
used and purchase sequential incranrtents of timely, useful information on 
program promise and perfontonce." (p. xiii) 

The four steps described are: (1) evaluability assessment; (2) feed- 
back evaluation; (3) performance monitoring; and (4) intensive evaluation. 
Evaluability assessment explores the expectations and information needs of 
policy makers and program managers and assesses the extent to which evalu- 
ation information is likely to be used by program management. The products 
of evaluability assessment are: (1) a set of agreed-on program objectives 
on which the program can realistically be held accountable; and (2) a set 
of evaluation/management options which represent ways in which management 
can change program activities, objectives, or uses of information in ways 
lil^aly to improve program performance. 

After the results of the evaluability assessment are completed, rapid 
feed-back evaluation surrmarized readily-obtainable information on program 
performance in tenns of those objectives. During this phase the evalu^.tor 
estirates the cost and value of additional information and presents designs 
for one or more full-scale evaluations. Step 3 is perfontance monitoring 
which meas\ires program perfontance in terms of the agreed-on objectives 
and oarpares program perfornaxx^e with prior or expected performance. In 
the final step, intensive evaluation , comparison or control groups are 
used to estimate the extent to which program results were caused by 
program activities. Each component in this "sequential purchase of 
information" is described with exairples, a description of important steps, 
and a discussion of the problems that occur for evaluators. 

The book os 227 pages (softoover) in double-space typed format. The 
style is parsimonious and conveys the infonration with little aifcellish- 
ment. The book is liberally illustrated with flow diagrams and inserted 
exairples. For exairple. Table I-l lists "Questions that evaluators 
typically face," including such items as "How do I define the purpose of 
this evaluation project?"/ "How do I determine when evaluation is not 
possible?", and "How can the evaluation process be structured so that the 
results are fed into the decision process and used?" Table II-5 shows a 
"Typical evaluability assessment site visit schedule/" including the 
activities scheduled frcm 9:00 a.m. of the first day to 4:00 p.m. of the 
second. Table II-6 gives a guide for interviews with local project 
staff. The appendix addresses such issues as "Keyzsteps in planning a 
sartple survey," and "Selection of sarrple design and sample size." 
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Although the book is written in a hi^y readable style, it is not 
read easily from beginning to end sinply because, as with nost evaluation 
issues, the detail and content build up quite rapidly. It requires effort 
by the reader to go throu^ the exaitples provided and elicit the mBaning 
and possible applix^ation bo another setting such as rehabilitation. No 
space is spent an discussing oonparative definitions or relating the 
discussion to theoretical nodels. ihe book is based instead on distill- 
ations fram the vork done by the Urban Institute, and those sources are 
quoted extensively. The bibliogr^y extends to seven pages and includes 
a wide range of sources not usually reviewed by program evaluators in 
rehabilitation. 



Evaluation; Promise and Performance is probably not the first book 
would want to read on evaluatimi it would appear to be of most 
interest to evaluators v*d have already tried to conduct evaluation in an 
a gency setting. It should help those vto are seeking a systematic 
approach in trying to deal with problems that seem to far exeed their 
available evaluation resources, and vto have already accepted the fact 
that there is a great amount of tedious detailed study required to 
understand and untangle the oonplexities of modem bureaucratic institu- 
tions. Our unit found it to be a refreshing, realistic, and powerful 
approach that we think will assist us to develop our skills as evaluators. 

The book is available from the publication office of the Urban 
Institute, 2100 M Street, N.W., Washington, D.C. 20037, for $7.50. 



BOOK EEVIEW 



TOE PRO FESSICN AND PRACTICE PIOGRAM EVALUATION 
SCAF^/IA B. ANEER5CN & SAMUEL BALL 



Reviewed by 

W.H. Brownfield, Virginia Department of Rehabilitative Services 



A program evaliaation primer! As the flysheet notes, the book is a pre- 
cise and practical guide to program evaluation , and offers as well, 
discussions of the numerous issues facing progr a m evaluation as an expanding 
profession. My reading of the book led me to agree that it is, incJeed, a 
practice-oriented approach focusing on day-to-day evaluation issxies rather 
than dealing with the theoretical debates. 

Following a discussion of issues in the emergir^ profession of PE, the 
book is divided into three sections vMch address (1) evaluation practices, 
(2) ethics and valuas in evaluation and (3) the future of program evaluation. 
Certainly this is not a specific "how to" evaluation text, nor does it pro- 
vide a oonprehensive discvission of research designs or statistical techniques 
useful in PE. On the other hand, it is almost "required reading" for any new 
staff mesnober joining a program evaluation unit or section. It seems peirticu- 
larly suited to new evaliaators in the vocational rehabilitation field or other 
VR staff interested in evaluation. Many of these persons cone to the staff 
assignment with much prograrmatic data related to the rehabilitation process 
but with little imderstanding of evaluation as either a practice or a pro- 
fession. 

Ihis 242-page book was published in 1978 by Jossey-Bass Publishers of 
San Francisco and is available at $13.95 a copy. 
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San Francisco: Jbssey-Bass, 1978 

Attkission, c. c,, et al. (Eds,) Evaluation of Hu rtan Service Proqrams, 
New York: Academic Press, 1978 ' ' 

Bernstein, I. N. & Freenan, H. E. Acadanic and Entrepreneurial R esearch- 

The Qonsequences o f Diversity in Federal Evaluation St u^ New York: 
Russel Sage Foundation, 1975 — 

Bolton, B. Handbook of Measurement and Eval uation, Balturore: University 
Park Press, 1976 
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National Training and Deveioptent Service Press, 1978 
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York: Russel Sage, 1977 " 
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Rehabilitatia i, Arkansas: Arkansas Rehabilitation Research and Train- 
mg Center, 1978 

Ctook, T, D, , & Reichardt, C, S. Qualitative and Quantitative Methcx3s in 
Evaluation Research , Beverly Hills: Sage Publications, 1979 

Ooursey, R, D. (Ed,) Program Evaluation for Mental Health: Methpds, Stra- 
tegies, and Participants . New York: Grune & Stratton, 1977 

Davidoff, I., Qattentag, M,, & Offut, j, (Ed,) Evaluating Obnnrunity Mental 
Health Services: Principles and Practice , Washington, D, C, : U,S, 
Govemnent Printing Office, 1978 

Epstein, I,, & Tripodi, T, Research Techniques for Program Planning, 

Mpnitoring and Evaluation , New York: Oolunibia University Press, 1977 

Fitz-Gibfaon, C. T,, & Morris L, L. How to Design a Program Evaluation , 
Beverly Hills, California: Sage Publications, 1979 

Fitz-Gibbon, C, T., & Morris, L, L, Evaluators Handbook , Beverly Hills: 
Sage Publications, 1978 
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Hafierson, M. E., Lyons, L. M., & Fitz-GiJion, C. T. Kow Tto Measure Attitu des, 
Beverly Hills: Sage Publications, 1978 

Harrisor, A. W., Jr. Evaluation In Legislation . Beverly Hills: Sage 
Publications, 19 

Manner, j. v. Ihe Process of Progran Evaluation , Washington, D.C. : National 
Training and Development Service Press, 1978 

Perloff, R. Evaluator_Interventions: Pros and Pons . Beverly Hills: Sage 
Publications, 1975 

Rich, R. F. Translating Evalxaation into Policy (Vbl. 3) . Beverly Hills: 
Sage Publications, 1975 

Itobinault, I. P. Program Planning 6f Evaluation; Selected Ttopics for 
Vocational Rehabilitation . 

Itossi, P. N., Freanan, H. E. & Wright, S. R. Evaluation; A Systenatic 
^^yroach . Beverly Hills; Sage Publications, 1979 

Schulberg, H. C. , Sheldon, A., Baker, F. Program Evaluation in Health 
Fields . New York: Behavioral Publicatuons, 1979 

Sze, W. C, Hopps, J. G. Evaluation and Acoountability in Human Service 
Programs, Secoxxa Edition . Cambridge, Massachusetts: Schenkman 
Publishing Qxrpany, 1978 

Suchnan, E. Evaluative Research . New York: Russell Sage Foundation, 1967 
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APPENDIX II 



PERIODICftIS AND JOUPNAIS 
RELATING TO REHABIIITmCN AND EVZaUATION 

Pfxril 1980 
REHABIIITATICN-REIAIED PEKEC»ICALS 
Anerican Rehabilitaticai 

Publisher: Rehabilitation Services Adrninistration (RSA) , EE 
Bi-monthy publicaticai 

Business Office: 

Si:perintendent of Docunents 
P.O. Box 1533 
Washington, D.C* 20402 

Editor; 

Rehabilitation Services ADministration 
330 "C" Street, S.W. 
Washington, D.C. 20201 



Informer 

PubUsher: Arkansas Rehabilitatiorx I^search & Training Center (Rr-13) 
Quarterly Publication 

Business Office: 

Arkansas Reseaxrh 

Aricansas Rehabili^tion Research & Training Center 
P.O. Box 135r 

Hot Springs. Arkansas yOl 

Editor: 

Same 



Journal of ;^>Fa.ied Rehabilitation CbTjnseiir^ 

PubUsher: National Rehabilitatioi Cbiaiseling Association 
Quarterly Publication 

Business Office: 

National Rehabili tation Association 
1522 "K" Street, W.W. 
Washington, D.C. 20005 
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Editor: 

University of Texas at Austin 
440B Education Building 
Austin, Texas 78712 

Journal of Rehabilitation 

Publisher: National Rehabilitation Association 
Qucu±erly Publication 

Business Office: 

National Rehabilitation Association 
1522 "K" Street, N,W. 
Washington, D.C. 20005 

Editor: 

Dr. Jack M. Sink 
University of Georgia 
413 AderhDld 
Athens, Georgia 30602 



Jbum^i of Rehabilitation Administration 

Publisher: Hie Rehabilitation Services Training Program of DePaul University 
Quarterly Publication 

Business Office: 

Administration Stiadies Center 
DePaul University 
25E Jackson Boxjievard 
Chicago, Illinois 60604 

Editor: 

Same 



Nstioiz^l Rehabilitation Oounseling Association ; P rofessional Report 

Publisher: National Rehabilitation Counseling Association 
Bi-itiDTri-iily Publication 

I'usiness Office: 

1522 "K" Street, N.W. 
Washington, D.C. 20005 

Rehabilitation Oounseling Bulletin 

Pi±>lisher: Arerican Personnel and Guidance Association 
Quarterly Publication 

Business Office: 

IVjo Skyline Place, Suite 400 

5203 Leesburg Pike 

Falls Church, Virginia 22041 



Editor: 

Rehabilitation Counseling Proqram 
Christopher Baldy Hall 416 
Suny at Buffalo 
Anherst, New York 14260 

Rehabilitation Gazette 

Business Office: 

Rehabilitation Gazette 

4502 Maryland Avenue 

St. louis, Missouri 63108 

Editor: 

Same 

Rehabilitation Literature 

publish^: National Easter Ssal Society for Crippled chUdren . Mults 

Business Office: 

2023 West Ogden Avenue 
Chicago, Illinois 60612 

Editor: 

Sane 

Rehabilitation Wbrld 

Publisher: Rehabilitation International USA 
Quarterly Publication 

Circulation Department 
Rehabilitation Department 
20 West 40th Street 
New York, New York 10018 

EVAIXJATICN-RELATED PERIODICftIS 

Evaluation 

Pu^ish^r Minneapolis Medical Research Ibundation, Inc. 
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circulation Manager 
Evaluation 

501 South Park Avenue 
Minneapolis, Minnesota 55415 



Evaluation Review ; A .Jovimal of Applied Social Research 

Publi-sher: Sage Publications 
Quarterly 

Business Office: 

Sage Publications 

275 South Beverly Drive 

Beverly Hills, California 90212 

Editor: 

Richard A. Buck 
Department of Sociology 
University of California 
Santa Ba3*ara, California 93106 



Measurement and Evaluation in Guidance 

Publisher: American Personnel and Guidance Associaticxi 
Quarterly Publication 

Business Office: 

TX/go Skyline Place 

Suite 400 

5203 Leesburg Pike 

Falls Churchr Virginia 22041 

Editor: 

Dr. Larry loesch 
College of Education 
University of Florida 
Gainesville, Florida 32611 



Educational Evaluation and Polic y Analysis 

Publisher: Anerican Educational Research Association 
Bi-monthly Publication 

Business Office: 

1230 17th Street, N.W. 
Washington, D.C. 20036 




BditDr: 

W. J. Popham 
S. M. W. Kirst 

Evaluation and the Health Profession ; A ^^Iltiprofessional Journgi 

Publisher: Sage Publications 
Quarterly Publication 

Business Office: 

Sage Publications 

275 South Beverly Drive 

Beverly Hills, California 90212 



R. B, Bansell 
C. A* Waltz 
School of Nursing 
University of Maryland 
655 West Lattbard 
Baltinore, Maryland 20201 



Publisher: Research and Evaliaation Services 
Quarterly Publication 

Business Office: 

Program on Press 

Pairview Park 

Elmford, New York 10523 

Editor: 

J. A. MDrrell 

Department of Health Sciences 
Hahnenann Medical College 
112 N. Broad Street 
Philadelphia, Pennsylvania 19102 

Ho^l^ PQlicy Quarterly : Evaluation and IH-iUzation 

Publisher: Human Sciences Press 
Quarterly Publication 

Business Office: 
H. S. P. 
72 5th Avenue 
New York, New York 10011 

Editors: 

H. C. Schulberg 
E. Ricci 



Editors: 



Evaluation and Program Planning 
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ADVISORY OCMHTIEE 

MOEEL EVALUATIGN 
UNITS PRQ3ECT 



tepresentatives f ran Rehabilitaticxn Services Administration 

RDbert L. Davis, Director, Division 

of Resource Jfenagement 
Department of Education 
330 "C" Street, S.W. 
Itocm 3230 

Washington, D.C. 20201 (202) 245-0085 • 

Wesley W. Geigel, Director 

Division of Rehabilitation Facilities 

Rxm 3329-S 

I^fery E. Switzer Building 
' 330 "C" Street, S.W. 

Washii^n, D.C. 20201 (202) 245-0076 

lawrence I. Mars 
Statistician & Leader of the 

Data Analysis Team 
Mary E. Switzer Building 
RDom 3117-S 
330 "C" Street, S.W. 

Washington, D.C. 20201 (202) 245-0080 

William Massie, Ocx)rdLnator 
Innovation & E)q>ansion Grant Program 
Rxxn 3418-S 

Mary E. Switzer Building 
330 "C" Street, S.W. 

Washington, D.C. 20201 (202) 245-0316 



Regional Office - Rehabilitation Services Administration 

Harold D. Viaille, Ph.D. 

Acting RSA Regional Program Director 

Department of Education 

Ifegional Office for Rehabilitation Services 
Region VI 

1200 Main Tower Building, Suite 2040 
Dallas, Itexas 75202 (214) 767-2961 
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Frank Canitell 

Office of Rehabilitation Services 

Department of Education 

Regiai X 

Arcade Building 

1321 Second Aveiue (MS505) 

SeatUe, Washington 98101 (206) 442-5331 



CSAVR 



Group 



John M. Cobun, Ph.D., Director 
Assistant State Superintendent in 

Vocational Itehabilitation 
State Department of Education 
P.O. Box 8717 

Baltimore, Maryland 21240 (301) 796-8300 

Kobeii: W. Dingwall, Bd.D. 
Director, Division of Vocational 

Rehabilitation 
De^rtanent of Health & Social Services 
Rocm 327, Hathaway State Office BuUdinq 
Cheyenne, Wyoning 82002 (307) 777-7389 

Mr. Terry Etling 

I>eputy Administrator 

Ohio Rehabilitation Services Caimission 

4656 Heaton Road 

Columbus, Ohio 43229 (614) 438-1220 

I^owell E. Green 

Admijiistrative Assistant, Rehabilitative 

and Visual Services 
Public Welfare Cdmnission 
Department of Institutions, Social 

and Rehabilitative Services 
Sequoyah Manorial Office Building 
P.O. Box 25352 

Oklahoma City, Oklahoma 73125 (405) 521-2907 



Regional R ehabilitation Research Institute 

Don Harrison, Ph.D. 
Associate Professor & Director 
University of Michigan 
Rehabilitation Research Institute 
1323 School Education Building 
610 E. Lfriiversity 

Ann Arbor, Michigan 48109 (313) 753-4795 
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Research Utilization Taboratory-ICD Rehabilitation & Research Center 

Mr. Marvin Weisinger 
Asscx:iate Si^^'visor 
RUL-ICD 

340 East 24th Street 

Nev; York, New York 10010 (212) 679-0100, Ext. 282 or 288 



Oklahcina Rehabilitation Administration & Managerrent Program 
Mr. L. H. Autry 

Director of Rehabilitation Administration 

and Management Programs 
University of Oklahoia 
1700 ASP Avenue, Building 158 
Norman, Oklahcraa 73073 (405) 325-4913 



Research Utilization Specialist 

Ihcmas E. Backer, Ph.D., Senior Staff Associate 

Human Interaction Research Institute 

10889 Wilshire Boulevard, Suite 1120 

Los Angeles, California 90024 (213) 879-1373 



Consumer Representative 

Ms. Kathy Miller 

Deputy Director 

Center of Handicapped Affairs 

1026 East Michigan Avenue 

Lansing, Michigan 48912 (517) 483-5887 



RSA Managanent Information System Representative 

Ron Geddes 
ABT Associates 
55 Wheeler Street 

Cambridge, Massachusetts 0213L (617) 492-7100 



Contract Representatives 

Richard Dodson, Ph.D. 
Berkeley Planning Associates 
1912 Bonita Avenue 

Berkeley, California 94704 (415) 549-3492 
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National Association of Rehabilitation Facilities 
5530 Wisconsin Avenue 
Suite 955 

Washington, D.C. 20015 (301) 654--5882 

Gary Prazak 
Walker & Associates 
123 East Grant Street 

Minneapolis, Minnesota 55403 (612) 870-4420 

Richard A. Nida, Ph.D. 

West Virginia Research & Training Center 

#1 IXinbar Plaza, Suite S 

IXinbar, west Virginia 25064 (304) 766-7138 



Project Officers 

James E. Taylor, Ph.D. 
Project Officer 

Coordinating .& Nbdel Unit Contract 
Room 3529, MES 
Department of Education 

Washington, D.C. 20201 (202) 245-3379 

Harold Kay, Ed.D. 
Assistant Project Officer 
ODordinating & Model Unit Contract 
Task II 

Office of Human Development 
Office of Rehabilitation Services 
3535 Market Street 
P.O. Box 13716 

Philadelphia, Pennsylvania 19101 (596-1330 




Ms. Anita McQuillen 

Co-Project Director on the Facilities Reporting 

Syst'Hn Contract 
Departnent of Education 
330 "C" Street, S.W, 
Ma-^^ E. Svdtzer Building 
Room 3327 

Washir a, D.C. 20201 (202) 234-0077 



Model Units 

Guy Veach 

Division of Vocational Rehabilitation 
820 N. French Street 

Wilmington, Delaware (302) 571-2874 

Robert Struthers, Ph.D. 
State of Michigan 
Department of Education 
Vocational Rehabilitation Services 
P.O. Box 30010 

Lansing, Michigan 48909 (517) 373-3970 

Wendell Cox 

Vocational Rehabilitation for the Blind 
P.O. Box 4872 

Jackson, Mississippi 39216 (601) 354-7704 

Neil Sher\rood 

Vocational Rehabilitation Division 
Department of Human Resources 
2045 Silverton itoad, N.E. 

Salem, Oregon 97310 (503) 378-3830 

William Jenkins, Ed.D. 

Bureau of Vocational Rehabilitation 

Labor & Industry Building 

7th and Forster Streets 

Harrisburg, Pennsylvania 17120 (717) 78'7'-7312 
W. H. Brownfield 

Department of Vocational Rehabilitation 
4901 Fitzhugh Avenue 
P.O. Box 11045 

Riclrond, Virginia 23230 (804) 257-0271 
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